Methods and apparatuses for controlling a system via a sensor

ABSTRACT

Embodiments include a method. The method includes maintaining, by a processing device, a real-time context describing a circumstance affecting an object. The method also includes determining a first saturation level of the object within a first portion of the field of view (FOV) of a sensor at a first point in time. The method also includes determining a second saturation level of the object within a second portion of the FOV of the sensor at a second point in time. The method also includes executing, by the processing device, an action to address a positive saturation determination between the first saturation level and the second saturation level.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.16/162,041, filed 16 Oct. 2018 which is a continuation of U.S. patentapplication Ser. No. 14/712,826, filed May 14, 2015, which claimspriority to U.S. Provisional Application No. 62/010,966, filed on Jun.11, 2014, which are hereby incorporated by reference for all purposes.

BACKGROUND

Sensors may be used to acquire data that is analyzed in terms ofcontent. For example, an image sensor may obtain an image, and thatimage then may be evaluated to detect features such as hand gestures. Itmay be possible to use data obtained in such fashion to control asystem. A hand gesture captured by an imaging sensor may be used as asystem input, e.g. to cause some processor command to be executed inresponse to the gesture, to control a device thereby, and so forth.

Such inputs may rely on gestures that are conveniently executed,distinct, “natural”, etc. However, the use of such “natural” gestures inthis fashion may pose problems. For example, multiple systems may relyon similar or identical natural gestures for conflicting purposes, e.g.one gesture (or two similar gestures) may be selected for differentinputs by developers of an operating system and an application that runsunder that operating system. This may produce confusion both for systemusers and within the system itself.

Furthermore, a gesture that is “natural” may be used without beingintended as a system input. For example, hand motions associated withthe manipulation of physical objects, hand motions made unconsciouslyduring a conversation, etc. may be made incidentally rather than asinput for entering commands, controlling systems, etc., and at leastpotentially may be interpreted erroneously as commands to a processingsystem.

BRIEF SUMMARY

The present embodiment contemplates a variety of systems, apparatus,methods, and paradigms for controlling a system via a sensor.

In one embodiment of the present embodiment, a machine-implementedmethod is provided that includes establishing a saturation profile in aprocessor including saturation of a sensor and establishing a saturationresponse in the processor including an executable instruction for theprocessor. The method includes sensing an input with the sensor,communicating the input to the processor, and comparing the input to thesaturation profile in the processor. The method further includes, if theinput satisfies the saturation profile, executing the saturationresponse in the processor.

The saturation profile may include at least one input feature inaddition to the saturation.

The sensor may have a field of view. The saturation profile may includesaturation of the sensor across at least a substantial portion of thefield of view thereof. The saturation profile may include saturation ofthe sensor across substantially all of the field of view.

The saturation of the sensor may substantially correspond with the endeffector.

The saturation of the sensor may substantially correspond with themaximum sensor value, the minimum sensor value, an invalid sensor value,an error sensor value, and/or a substantially uninformatively uniformsensor value.

The input may include an image. The input may substantially correspondwith a hand covering substantially all of the field of view of the imagewith a minimum brightness. The saturation of the sensor maysubstantially correspond with maximum brightness, minimum brightness,maximum color channel brightness, minimum color channel brightness,substantially uninformatively uniform brightness, substantiallyuninformatively uniform color channel brightness, invalid brightness,invalid color brightness, error brightness, and/or error colorbrightness.

The input may include a depth image. The input may substantiallycorrespond with a hand covering substantially all of the field of viewof the depth image with a minimum depth. The saturation of the sensormay substantially correspond maximum depth, minimum depth, substantiallyuninformatively uniform depth, invalid depth, and/or error depth.

The input may include depth data. The saturation of the sensor maysubstantially correspond with maximum depth, minimum depth,substantially uninformatively uniform depth, invalid depth, and/or errordepth.

The input may include audio data. The saturation of the sensor maysubstantially correspond with maximum volume, minimum volume, maximumfrequency volume, minimum frequency volume, substantiallyuninformatively uniform frequency distribution, invalid input, and/orerror input.

The input may include thermal data, ultrasonic data, time of flightdata, stereo depth data, focal depth data, accelerometer data, gyroscopedata, electrical data, and/or magnetic data.

The saturation profile may include a posture of an end effector and/or agesture of the end effector. The end effector may be a hand. The postureand/or gesture may include the plane of the hand being substantiallyflat to the field of view of the sensor. The saturation of the sensormay include the end effector being disposed so as to fill substantiallyall of the field of view.

The executable instruction may include a system command for theprocessor. The executable instruction may include a system interfacecommand. The executable instruction may include a “go back” command,wherein the system substantially returns the state of the interface to aprevious state of the interface.

The saturation profile may include a gesture terminating in thesaturation of the sensor. The gesture may include a hand in a field ofview of the sensor, the plane of the hand being substantially flat tothe field of view, the fingers of the hand being substantially extendedand at least partially spread. The saturation of the sensor may includethe hand being disposed so as to fill substantially all of the field ofview.

The saturation profile may include a gesture originating in thesaturation of the sensor. The gesture may include a hand in the field ofview of the sensor, the plane of the hand being substantially flat tothe field of view, the fingers of the hand being substantially extendedand at least partially spread. The saturation of the sensor may includethe hand being disposed so as to fill substantially all of the field ofview.

The saturation profile may include a gesture with the saturation of thesensor intermediate therein. The gesture may include a hand in the fieldof view of the sensor, the plane of the hand being substantially flat tothe field of view, the fingers of the hand being substantially extendedand at least partially spread. The saturation of the sensor may includethe hand being disposed so as to fill substantially all of the field ofview.

In another embodiment of the present embodiment, an apparatus isprovided that includes means for establishing a saturation profilecomprising saturation in an input, and means for establishing asaturation response. The apparatus includes means for sensing the input,means for comparing the input to the saturation profile, and means forexecuting the saturation response if the input satisfies the saturationprofile.

In another embodiment of the present embodiment, an apparatus isprovided that includes a sensor and a processor in communication withthe sensor. The apparatus includes a saturation profile instantiated onthe processor and a saturation response instantiated on the processor.The apparatus also includes a saturation profile comparer instantiatedon the processor and adapted to compare an input from the sensor withthe saturation profile so as to determine whether the input satisfiesthe saturation profile. The apparatus further includes a saturationresponse executor instantiated on the processor and adapted to executethe saturation response if the input satisfies the saturation profile.

The sensor and processor may be disposed on a head-mounted display.

The sensor may include an imager, a stereo image pair, a depth imager, adepth sensor, an audio sensor, an ultrasonic sensor, a thermal sensor, atime of flight sensor, a focal depth sensor, an accelerometer, agyroscope, an electrical sensor, and/or a magnetic sensor.

BRIEF DESCRIPTION OF THE DRAWINGS

Like reference numbers generally indicate corresponding elements in thefigures.

FIG. 1A shows an example field of view in comparison to a hand.

FIG. 1B shows another example field of view in comparison to a hand,with the hand substantially completely filling the field of view.

FIG. 1C shows another example field of view in comparison to a hand,with the hand filling a substantial portion of the field of view.

FIG. 2 shows an example method for delivering input to a systemaccording to the present embodiment, in flow-chart form.

FIG. 3 shows another example method for delivering input to a systemaccording to the present embodiment, with reference to a head-mounteddisplay, in flow-chart form.

FIG. 4 shows an example method for implementing steps for deliveringinput to a system according to the present embodiment onto a processor,in flow-chart form.

FIG. 5 shows an example apparatus for delivering input to a systemaccording to the present embodiment, in schematic form.

FIG. 6 shows an example apparatus for delivering input to a systemaccording to the present embodiment, in perspective view.

FIG. 7 shows a block diagram of a processing system that may implementoperations of the present embodiment.

DETAILED DESCRIPTION

Referring to FIG. 1A, therein a hand 102A is shown in outline form. Arectangular outline representative of a field of view 104A is shown torepresent the field of view of a sensor, such as a camera, depth sensor,etc. As may be seen, in the configuration shown in FIG. 1A the hand 102Aoccupies only a portion of the field of view 104A. A sensor with such afield of view 104A thus typically may distinguish and/or identify thepresence of the hand 102A, distinguish and/or identify postures of thehand 102A, distinguish and/or identify gestures executed by the hand102A, etc.

Turning to FIG. 1B, therein a hand 102B is again shown in outline form,and a rectangular outline representative of a field of view 104B isagain shown to represent the field of view of a sensor sensing (or atleast adapted to sense) that hand 102B. As may be seen in theconfiguration shown in FIG. 1B the hand 102B occupies the entire fieldof view 104B. A sensor with such a field of view 104B may be unable todistinguish and/or identify the hand 102B, distinguish and/or identifypostures of the hand 102B, distinguish and/or identify gestures executedby the hand 102B, etc., or at least such actions may be problematic.Rather, such a sensor typically would sense a substantiallyundistinguished data set throughout the field of view 104B; while thatdata set may indeed represent the hand 102B, the data set may not besufficient as to enable determining even that the hand is present, muchless the posture/gesture of the hand, etc.

An arrangement such as that in FIG. 1A may occur when a sensor is atsome significant distance from a hand 102A (or other entity), thedistance being suitable as to facilitate discrimination of the hand by asensor. In colloquial terms, the contents of the field of view 104A aresuch that the sensor would be able to tell that a hand 102A was present.More precisely, interpretation of the sensor input from the sensor, forexample in a processor, may be enabled.

By contrast, an arrangement such as that in FIG. 1B may occur when asensor is so close to a hand 102B (or other entity) that the field ofview 104B does not encompass enough of the hand 102B to enablediscrimination thereof. Even if some features could be detected oridentified, e.g. skin color, depth map, etc., identification of the hand102B on the basis of what is sensed may not be feasible, or may even beimpossible. It may be considered that for the arrangement in FIG. 1B theinput available to and/or provided by an image sensor may no longer be auseful image (and arguably may not be an “image” at all in certainstrict senses, i.e. is an all-black or all-white “image” an image ofanything?), or at least no longer may be an image of the hand 102B inthe sense of the hand 102B being recognizable therefrom.

In terms of the sensor itself, in practice for certain targets such as ahand the sensor may be considered to be saturated, or at leastsubstantially saturated, for arrangements such as that shown in FIG. 1B.For example, a camera may be nearly or entirely “blacked out” by thehand 102B throughout all or nearly all of the field of view 104B, whilea depth mapping sensor may read zero or near-zero distance (or potentialerrors or some other such “value”) across the entirety of the field ofview. Although such conditions do not necessarily represent “all black”input and may include for example all (or substantially all) white,error, zero or one (for binary systems), etc., it may be convenient toaddress such an input as a “blackout”.

Such a “blackout” also may be referred to as a form of sensorsaturation. Indeed, for at least certain sensors such a blackout may inpractice include saturation of the sensor elements, e.g. an all-whiteimage may set all elements of a CCD to their maximum charge status.However, physical saturation is not necessarily required, and thepresent embodiment is not limited only to arrangements where physicalsaturation is present.

To more fully understand sensor saturation, consider an arrangement of ablack-and-white digital camera, each pixel in an image might have abrightness value ranging from 0 (black, or the minimum input that can besensed) to 255 (white, the maximum input that can be sensed). When thesensor is fully blacked out (e.g. if fully obstructed by a hand) thesensor detects values that are at or at least substantially at 0 acrosssubstantially all or all of the field of view, and the sensor may bereferred to as being “saturated low” in terms of image brightness.

When a sensor's field of view (or some portion thereof) is saturated insuch fashion, whether high or low (maximum value or minimum value), whatthe sensor reports may no longer be an “image” in practical terms. Thatis, while outputting “0 brightness at all pixels” might be argued tostill technically constitute delivering data, an all-0-brightness returnisn't an image of anything.

While an all-0-brightness image is used as an example, otherpossibilities exist for sensor saturation. For the same sensor as thatpresented as an example above, an all-255-brightness return (sometimesreferred to as a “white out” or “burn out”) likewise may representsaturation. For color imaging sensors, e.g. sensors with multiple colorchannels (rather than a single black-to-white channel as in the exampleabove), substantially maximum or substantially minimum values in one ormore color channels across a substantial portion of the field of viewmight constitute saturation.

Furthermore, even if sensors are not necessarily at maximum or minimumvalues (black or white, full blue or zero blue, etc.), saturation stillmay be considered to occur if substantially the entire field of view isat least substantially non-distinguishable. For example, if a field ofview is saturated with a particular flesh tone or even a range of fleshtones, so as to provide inadequate information to distinguish (in thisexample) hand features, the sensor and/or image still may be consideredto be saturated for at least certain embodiments of the presentembodiment. That is, even if sensor input is not a maximum value,minimum value, or an invalid or error sensor value, sensor values thatare sufficiently uniform or undistinguished as to be substantiallyuninformative (e.g. being insufficient to discriminate content thereof)still may be considered as saturation, and utilized thereas according tothe present embodiment.

In addition, although black-and-white and color imaging sensors arereferenced above for simplicity, different sensors may exhibit differentforms of saturation. For example, consider a depth camera such as onethat produces a two-dimensional image of depths or distances betweensensor and subject matter, e.g. by measuring time-of-flight. In anarrangement similar to that shown in FIG. 1B, where a hand 102B is soclose to a sensor as to substantially fully obstruct the field of view104B, the distance from the sensor to hand 102B might be so short as tobe unmeasurable by the sensor. This might produce an all-0-distancereturn, however, for certain sensors and under certain conditions, thesensor might fault rather than returning data (even 0-distance data).That is, the sensor may generate an image wherein the pixels have nodistance data at all (as opposed to indicating 0 distance). Alternately,certain sensors may, due to the particulars of their design,construction, programming, etc., return values that are nonsense orphysically impossible, such as a value of −1 for distance.

The particulars of what constitutes a saturation response will vary withsensors, applications, etc. Thus while substantially minimum return,substantially maximum return, and fault returns are presented herein asexamples, it should be understood that other saturation states may alsoexist and fall within the scope of the present embodiment.

Regardless of the particular form or nature of sensor saturation, sensorsaturation may in some sense be considered a “non-response” from thesensor. Though the sensor may be physically functional and may deliverinput (e.g. to a processor, a display, etc.), the data, state, etc.being delivered by the sensor conventionally may be ignored as notuseful.

However, even though a saturated sensor may not enable distinguishing animage (e.g. sensing a hand and/or identifying gestures/postures) in thearrangement of FIG. 1B, the saturated sensor state nevertheless may beutilized as useful input according to the present embodiment. The factof the sensor saturation, and/or the particulars of the sensorsaturation (e.g. saturated black, saturated with a specific color orcolor range, saturated with distance faults, etc.) may be interpreted asan indication of some state, event, etc. For the example arrangementshown in FIG. 1B, sensor saturation may be interpreted as an indicationthat a hand 102B has approached the sensor to a distance such that thefield of view 104B is substantially filled and the sensor input issaturated by the input received from the hand 102B Likewise,circumstances of the sensor saturation also may be interpreted usefullyaccording to the present embodiment. For example, if previous input fromthe sensor showed a hand 102B approaching the sensor, this also may beused to interpret the sensor saturation as being representative of thehand 102B having closely approached the sensor to the point of sensorsaturation.

A saturation event thus may be utilized as an input according to thepresent embodiment, for example for a system engaged with the sensor,such as a processor, even if the sensor is producing no distinguishableimages.

Reference to image and/or image sensors is an example only, and otherarrangements may be equally suitable; the present embodiment is notnecessarily limited only to images and/or image sensors. For example,saturation of audio sensors with the noise of high volume, low volume,uniform pitch, indistinguishably uniform content (i.e. “white noise”),and so forth also may be suitable for certain embodiments of the presentembodiment. As a more concrete example, tapping a microphone or otheraudio sensor may produce a temporary saturated-high state, that is, themicrophone may read maximum (or at least high) input due to the tap.Conversely, covering a microphone so as to partially or entirely mufflesound incoming thereto may produce a saturated-low state, wherein themicrophone may read zero/minimum (or at least low) input thereby.Furthermore, noises produced without necessarily physically interactingwith an audio sensor also may produce saturation, for example byclapping, snapping fingers, slapping one's forehead with an open palm,etc. to yield a saturated-high state in the audio sensor.

A thermal sensor may be saturated by covering that sensor with a hand,in a manner potentially similar to the approaches described already withregard to image sensors. Whatever the temperature of the hand (typicallythough not necessarily high), covering a thermal sensor therewith mayproduce a saturated state with the thermal sensor at maximum or highlevels, a saturated state with the thermal sensor detecting thermalinformation sufficiently uniform as to be uninformative, etc.

Other sensors also may be suitable for use with the present embodiment,including but not limited to distance sensors (such as ultrasonicsensors, time of flight sensors, stereo depth sensors, focal depthsensors, depth cameras, etc.), motion sensors (such as accelerometers,gyroscopes, etc.), and electrical and/or magnetic sensors. Othersensors, and/or other saturation states, may be equally suitable. Inaddition, the particulars of what saturation states may be attained fora given embodiment may depend at least in part on the sensor(s)associated with those embodiments, and saturation states other thanthose described herein also may be equally suitable.

For an arrangement wherein postures and/or gestures are used as input,and with sensor saturation utilized as input as described with regard toFIG. 1A and FIG. 1B, “saturating the sensor” may be considered to be aposture or gesture (or if not then potentially at least some other formof user input). For example, a gesture that incorporates sensorsaturation may begin with a hand arranged with the palm flat to/facingthe sensor and with the fingers extended and spread, and transition tosaturation by moving the hand (and/or by moving the sensor) sufficientlyclose to the sensor that the sensor input is saturated. FIG. 1A and FIG.1B may be viewed as a simple saturation gesture conforming to thisexample description.

Consideration of sensor saturation with regard to inputs including butnot limited to posture/gesture inputs according to the presentembodiment may exhibit advantages. For example, although moving a handto saturate a sensor (e.g. moving a hand to cover the lens of an imagingsensor) may be a convenient and in at least some sense a “natural”movement to execute, such a motion may not be characteristic of gesturesused for non-input purposes (e.g. for emphasis in casual conversation).More colloquially, a user who is not wearing a camera generally does notcarry out motions so as to saturate the input of a nonexistent camera(thus such saturation gestures may not have common equivalents intypical person-to-person communication gestures, etc.). More concretely,considering as an example a head-mounted display in the form of a pairof glasses having cameras to either side, moving to obstruct one or bothcameras with a hand may not have an unconscious or casual analog, sinceabsent the cameras such motions may serve no purpose.

By contrast, a person in conversation may gesture, perhaps without evenbeing aware of gesturing. Similarly, certain common deliberate gesturesrelating to grabbing, manipulating, and moving objects, whilepotentially useful as inputs, are frequently carried out for purposesother than delivering an input to a system (e.g. in order to interactwith objects in the physical world). If such postures/gestures aresensed by sensors, such gestures may be interpreted by a system asinputs/commands regardless of the intent of the person making thosepostures/gestures. Such events may be referred to as “false positives”;the system receives and/or reacts to a command that the user did notintend to give.

Thus one advantage of saturation-linked postures and/or gestures may bea resistance to false positives, insofar as users may be unlikely toexecute saturation-linked postures/gestures unconsciously or foralternative purposes.

Another potential advantage of saturation-linked inputs according to thepresent embodiment may be that multiple variants of postures and/orgestures may be available. For example, substantially any hand positionand/or motion (or a position/motion of other end effectors, such as apen, a stylus, etc.), and/or other input (audio, etc.) may be combinedwith sensor saturation. Moving a hand 102A with fingers extended andspread from a position where the hand 102A may be distinguished by asensor as in FIG. 1A into a position where the hand 102B saturates asensor as in FIG. 1B may be suitable for use as a saturation-linkedgesture; however, the reverse—beginning by saturating the sensor andthen moving away—also may be suitable. Likewise, gestures that begin andend with saturation, that have saturation as a middle step, etc.likewise may be suitable. Furthermore, a wide variety of handconfigurations and/or motions may be suitable for portions of a gesturewherein the hand can be distinguished by the sensor, e.g. a closed fist,an extended index finger, a “Vulcan salute”, etc.

By contrast, a significant number of non-saturating gestures that may bepotentially suitable as inputs may be “off-limits” due to issues ofconfusion as noted above. For example, as noted above interpreting agrabbing gesture as a system input (e.g. grabbing a virtual object) maybe problematic since such a grabbing gesture also may be made when auser is actually grabbing a physical object.

It is noted that although different causes may produce sensorsaturation, not all sensor saturations are necessarily equivalent. Asensor saturation produced by holding a hand in front of a camera may bedistinguishable from a sensor saturation wherein hair or a hat blocks acamera, for example by color or other parameters. In addition, asaturation-linked gesture wherein saturation is to be followed by aparticular hand configuration (or stylus gesture, or other end-effectorconfiguration, etc.) may be unlikely to be misinterpreted; unless anunintended sensor saturation were followed by the user coincidentallyperforming the right-hand configuration at the right time, the fullgesture would not have been performed, so a spurious system command maynot have been executed.

Thus saturation does not in itself necessarily also introduce additionalfalse positives, and indeed in at least certain instances may contributeto avoiding false positives.

Now with reference to FIG. 1C, sensor saturation may manifest for only aportion of a sensor field of view; saturation is not limited only tofull-field-of-view effects, nor is the present embodiment limited onlyto full-field-of-view sensor saturation.

As may be seen in FIG. 1C, a hand 102C is again shown in outline form,and a rectangular outline representative of a field of view 104C isagain shown to represent the field of view of a sensor sensing the hand102C (and potentially delivering sensor input to a processor, etc.).Although the relative sizes of the hand 102C and field of view 104C aresimilar to those in FIG. 1B, in FIG. 1C the hand 102C occupies only aportion of the field of view 104C. For such an arrangement it may beuseful to distinguish two regions within the field of view 104C: a firstregion 106C of the field of view that is not saturated (or at least isnot saturated by the hand 102C; background conditions may in certaincircumstances cause saturation, e.g. direct sunlight for an imagesensor, but such issues are not represented or considered to be presentin FIG. 1C for purposes of simplicity), and a second region 108C of thefield of view that is saturated. Such an arrangement typically mayresult in the sensor sensing a substantially undistinguished data setthroughout the second region 108C, and (potentially) a distinguishable(“normal”) data set in the first region 106C.

An arrangement such as that in FIG. 1C might occur when a sensor is soclose to a hand 102C (or other entity) as to partially cover the sensor,leaving the second region 108C of the field of view 104C incapable ofdiscriminating the hand while the first region 106C may still captureinformation normally.

Turning now to FIG. 2 , therein is shown an example method forcontrolling a system according to the present embodiment, in flow-chartform. In the example shown in FIG. 2 , a saturation profile isestablished at step 212. Typically but not necessarily, the saturationprofile may be established as data and/or executable instructionsinstantiated on a processor, and/or stored so as to be accessible to theprocessor. The saturation profile may be understood to define and/ordescribe what constitutes sensor saturation.

For example, a saturation profile might specify that at least 95% of thefield of view must exhibit 0% to 2% brightness, i.e. a substantialblack-out in substantially the full field-of-view. Alternately, asaturation profile might specify that at least 20% of the field of viewmust exhibit depth fault returns, i.e. a partial depth sensor black-out.Saturation profiles may also include additional factors, for example, arequirement that regions (e.g. the 95% and 20% above) be contiguous,have a certain shape, have well-defined borders, etc.

The specifics of the saturation profile may vary depending on a varietyof factors, including but not limited to the type and performance of thesensor (e.g. a black-and-white camera typically will not haverestrictions regarding color) and the types of inputs that are expectedto produce saturation (e.g. black-out saturation by placing a hand infront of a depth sensor may be expected to generate a sensor feed thatis different from white-out saturation by illuminating an image sensorwith an LED on a stylus, etc.). Choices for different embodiments alsomay affect the details of the saturation profile, for example, asaturation profile for a sensor on a head-mounted display might bedefined to exclude saturation by the hair, hats, etc.

In addition, as noted above saturation-linked postures and/or gesturesmay include inputs other than the saturation itself, such as handconfigurations/motions before and/or after saturation. The saturationprofile thus may be defined so as to include non-saturation information,such as images of hand postures and/or gestures, and/or other inputs.

Furthermore, the saturation profile may be conditional, with differentrequirements for different conditions. For example, considering anarrangement wherein the present embodiment is implemented in ahead-mounted display, a saturation profile may be defined with arequirement that the head-mounted display must be worn (perhaps asdetermined through sensor input), and that the saturation profile wouldnot be satisfied under any conditions (or under very limited conditions)if the head-mounted display is not worn. For such an arrangement,placing the head-mounted display inside a pocket, purse, etc. would notthen necessarily trigger an unwanted command due to sensor saturation.

The present embodiment is not particularly limited with regard to thesaturation profile, and other arrangements than those examples describedmay be equally suitable.

Typically though not necessarily, the saturation profile may beestablished in a processor. In such embodiments, the present embodimentis not limited with regard to the processor. A range of general-purpose,special-purpose, and embedded systems may be suitable for use as aprocessor for the present embodiment. Moreover, it may be equallysuitable for the processor to consist of two or more physical or logicalprocessor components, or to be a “virtual” processor. Other arrangementsalso may be equally suitable.

With regard in particular to the term “establishing”, establishing thesaturation profile is to be understood broadly with regard to thepresent embodiment. It is noted that to “establish” something may,depending on particulars, refer to either or both the creation ofsomething new (e.g. establishing a business, wherein a new business iscreated) and the determination of a condition that already exists (e.g.establishing the whereabouts of a person, wherein the location of aperson who is already present at that location is discovered, receivedfrom another source, etc.). Similarly, establishing a saturation profilemay encompass several potential approaches, including but not limited tothe following.

Establishing a saturation profile may include acquiring an existingsaturation profile from some source, e.g. a data store such as a harddrive or solid state drive, a communicator such as a wired or wirelessmodem, information stored in and/or with a sensor (e.g. calibrationprofiles in read-only memory that may include “fault” conditions for afault saturation), etc.

Establishing a saturation profile also may include creating orcalculating the saturation profile, e.g. a processor may executeinstructions so as to determine a saturation profile computationally,for example considering the type of sensor, previous sensor input, etc.

Some combination of the above approaches for establishing a saturationprofile, and/or alternate approaches, may be equally suitable. Thepresent embodiment is not limited insofar as how a position may beestablished. So long as a saturation profile is in some manner madeavailable for the necessary functions thereof, any approach forestablishing the saturation profile may be suitable.

Similarly, the establishing of other features according to the presentembodiment (e.g. a saturation response) likewise should be understoodbroadly, and the present embodiment is not particularly limited withregard to the manner in which those features may be established unlessotherwise specified herein.

Continuing in FIG. 2 , a saturation response is also established at step214. Typically but not necessarily, the saturation response isestablished as data and/or executable instructions instantiated on aprocessor, and/or stored so as to be accessible to the processor. Thesaturation response defines and/or specifies actions that may be takenby the processor (and potentially by other entities in communicationwith the processor) in response to saturation.

For example, a saturation response may include the processor executingsome system command, performing some action within a user interface,etc. A saturation response may be defined as a fixed response, e.g. a“go back” command that substantially returns a user interface to aprevious state or condition at substantially any time and/or undersubstantially any conditions. However, the saturation response also maybe defined conditionally, such that different responses are executeddepending on differing conditions, e.g. “go back” under certaincircumstances, “help menu” under other circumstances, etc.

The present embodiment is not limited with regard to the saturationresponse, and other arrangements than those examples described may beequally suitable.

Still, with reference to FIG. 2 , the input is sensed in step 216. Inputincludes (but is not necessarily limited to) information that enablesdetermination of saturation, such as an image, video, depth map, etc.The present embodiment is not limited with regard to what form the inputmay take or how the input is sensed. Typically though not necessarily,the input may be sensed by a physical sensor proximate and/or integratedwith the processor, such as a camera, depth camera, ultrasonic system,or other imager disposed on an electronic device with a processortherein, such as a head-mounted display. However, this is an exampleonly, and the sensor is not required to be either integrated with orproximate the processor.

The present embodiment also is not limited with regard to the sensor,and a range of devices may be suitable for use as a sensor for thepresent embodiment. In certain examples presented herein the sensor isan imaging sensor, adapted to obtain still images and/or video. Suitableimaging sensors may include but are not limited to digital CMOS and CCDcameras. However, other sensors, including but not limited to depthsensors, ultrasound sensors, and sensors that capture information otherthan images and/or video may be equally suitable.

Continuing in FIG. 2 , the processor compares the sensor input to thesaturation profile at step 220. That is, information sensed by thesensor (in step 216) is compared at step 220 with the condition(s)established for the saturation profile (in step 212).

Based on the comparison of sensor feed and saturation profile in step220, a determination is made at step 222 as to whether the sensor feedsatisfies the saturation profile. If the determination is positive—ifthe sensor feed does satisfy the saturation profile—then the methodcontinues with step 224 (below). If the determination is negative—if thesensor feed does not satisfy the saturation profile—then the methodskips step 224.

Continuing in FIG. 2 , for instances wherein the determination at step222 is positive, the saturation response is executed at step 224. Thatis, the action(s) defined/specified in step 214 are carried out,typically though not necessarily by transmitting data from/through theprocessor, executing executable instructions in the processor, sendinginstructions to some device or system in communication with theprocessor, etc.

Although FIG. 2 shows the method therein as being complete followingstep 224 (if the comparison is positive, or step 222 if the comparisonis negative), it is emphasized that the method in FIG. 2 is an exampleonly. Other steps, other functions, etc. may be incorporated into themethod, and/or other methods may be executed in combination with themethod according to the present embodiment. In addition, for at leastcertain embodiments at least some portion of the method may repeat, e.g.in an ongoing loop that continues to determine whether the saturationprofile is satisfied by the sensor feed. This likewise applies to othermethods shown and described herein.

It is noted that as shown in FIG. 2 , a method according to the presentembodiment includes executing a saturation response if the saturationprofile is satisfied by the input. That is, the present embodiment isnot merely directed to the existence of saturation as a phenomenon, norto the sensing of saturation, nor even to determine whether saturationis present or meets some standard. Rather, the present embodimentcarries out some positive action in response thereto. Moreover, thatpositive response may perform some useful function, such as issuing asystem command to control a device. It is emphasized that according tothe present embodiment, sensor saturation is not merely considered as“waste” or as a problem, or even ignored as non-input, but rather sensorsaturation is applied to serve as a form of useful input in and ofitself.

Now with reference to FIG. 3 , where FIG. 2 presented a relativelygeneral description of a method according to the present embodiment,FIG. 3 describes a more concrete example. Notably, FIG. 3 refersspecially to a wearable electronic device that may be described as ahead-mounted display or HMD, and to elements and functions thereof. Thearrangement of FIG. 3 is an example only; the present embodiment is notlimited only to head-mounted displays, and other arrangements than thoseshown in FIG. 3 may be equally suitable.

In the example arrangement of FIG. 3 , a saturation profile isinstantiated at step 312 onto the processor of an HMD. The saturationprofile is adapted to address input from a depth camera disposed on theHMD, with the saturation profile specifying that the full field of view(FOV) of the depth camera should exhibit a distance value of 0.

Moving on in FIG. 3 , an HMD system command is instantiated at step 314on the HMD processor as a saturation response. That is, some command isspecified to be issued by the processor, in the event that thesaturation profile is satisfied by sensor input. For the example shown,wherein the processor is disposed within an HMD, the command mightaddress some other physical element of the HMD, such as instructing achange to the output of a display screen. For the purposes of theexample arrangement in FIG. 3 , the system command is considered to be acommand executed by the processor that controls some physical systemand/or behavior of the HMD itself. Thus, the method shown in FIG. 3represents a method of controlling an electronic device. However, thisis an example only, and other arrangements also may be suitable,including but not limited to commands addressing an operating system onthe HMD processor, programs on the HMD processor, an external system incommunication with the HMD, etc. Furthermore, commands controlling otherdevices and/or systems (electronic or otherwise) besides an HMD, andcommands not necessarily controlling a device or system, may be equallysuitable.

Input is sensed at step 316 with the depth camera disposed on the HMD.For a depth camera, typically though not necessarily such input may be adepth map or depth image, a two-dimensional array of pixels wherein eachpixel thereof has a depth or distance associated therewith (analogous tothe color values associated with pixels in a color digital image).However, other arrangements may be equally suitable.

The input is communicated at step 318 to the processor. (It is notedthat in FIG. 2 , wherein certain steps were not necessarily specified astaking place within a processor, nor input to have come from a sensor,the notion of communicating may be considered implied, given that themethod is not limited to a specific source or destination between whichcommunication would pass. Thus, no explicit analog of step 318 appearsin FIG. 2 .) For an HMD with the processor and sensor both disposedtherein, communication may take place along a direct wired link.However, this is an example only, other arrangements for communicationmay be equally suitable, and the present embodiment is not limited withregard thereto.

Still, with reference to FIG. 3 , the input is compared at step 320within the processor against the saturation profile. Typically thoughnot necessarily the comparison may be carried out by executableinstructions instantiated on the processor, though other arrangementsmay be suitable.

A determination is made at step 322, based on the comparison at step320, as to whether the saturation profile is satisfied by the sensorinput. For the specific example of FIG. 3 , the determination may bestated as: does the depth image (sensor input from step 316) exhibitzero distance for the pixels thereof, across the full field of view ofthe sensor (as established in step 312)? If so, then the method proceedsto steps 324 and 326. If not, then the method skips steps 324 and 326.

If the determination at step 322 is positive, then the HMD systemcommand is issued at step 324 by the processor (that command having beendefined as a saturation response in step 314). Moving on in FIG. 3 ,whatever system command is issued at step 324 by the processor iscarried out within the HMD, so as to control the HMD. As noted, such acommand may control screen outputs, program or operating systemfunctions, etc., and the present embodiment is not limited with regardthereto.

Turning to FIG. 4 , therein is shown an example method for disposingonto a processor an arrangement for carrying out a method forcontrolling a system according to the present embodiment, in flow-chartform. For the purposes of FIG. 4 , it is presumed that a sensor adaptedto sense input according to the present embodiment is in communicationwith the processor, so as to enable functions utilizing such a sensor(e.g. sensing input for a determination as to whether that inputexhibits saturation).

In the method of FIG. 4 , a saturation profile is instantiated at step432 onto the processor. Saturation profiles according to the presentembodiment have been described previously herein. A saturation responseis instantiated at step 434 onto the processor. Saturation responsesaccording to the present embodiment also have been described previouslyherein.

A saturation profile comparer is instantiated at step 436 on theprocessor. The saturation profile comparer is adapted to compare asensor input that may be received in the processor (e.g. from thesensor) with the saturation profile instantiated at step 432 on theprocessor. Typically though not necessarily the saturation profilecomparer includes executable instructions. However, other arrangementsmay be equally suitable, including but not limited to a saturationprofile comparer that includes independent dedicated hardware (though insuch instances the saturation profile comparer may be placed incommunication with the processor rather than being instantiatedthereon). Comparison of a sensor feed with a saturation profile has beenpreviously described herein.

A saturation response executor is instantiated at step 438 on theprocessor. The saturation response executor is adapted to execute thesaturation response if the saturation profile comparer determines thatthe sensor input satisfies the saturation profile instantiated at step434 on the processor. Typically though not necessarily the saturationprofile comparer includes executable instructions. However, otherarrangements may be equally suitable, including but not limited to asaturation response executor that includes independent dedicatedhardware (though in such instances the saturation profile comparer maybe placed in communication with the processor rather than beinginstantiated thereon). Comparison of a sensor feed with a saturationprofile has been previously described herein.

Typically though not necessarily the saturation profile, saturationresponse, comparer, and executor as referenced with respect to FIG. 4may be instantiated onto the processor from a data store such as a harddrive or solid state drive, or received by communication such as from anexternal device or network. The present embodiment is not particularlylimited with regard to how the saturation profile, saturation response,comparer, and executor are instantiated, or the source(s) from which thesaturation profile, saturation response, comparer, and executor areinstantiated.

Now with reference to FIG. 5 , therein is shown an example embodiment ofan apparatus 550 according to the present embodiment, in schematic form.The apparatus 550 includes a processor 552 and a sensor 554 incommunication therewith. Although the apparatus 550 depends to someextent on sensor input, the sensor 554 itself need not necessarily bephysically part of the apparatus, nor in direct communication with theprocessor 552, so long as a sensor input therefrom is available to theprocessor 552. Thus a remote sensor, data recorded at some other time bya sensor, etc. may be equally suitable, though for simplicity a sensor554 is shown in FIG. 5 as part of the apparatus 550 proper.

The apparatus 550 also includes a saturation profile 556, a saturationresponse 558, a saturation profile comparer 560, and a saturationresponse executor 562 instantiated thereon. A saturation profile 556,saturation response 558, saturation profile comparer 560, and saturationresponse executor 562 according to the present embodiment have beendescribed previously herein.

The present embodiment may be used with and/or incorporated into a widevariety of other devices, and may take a wide variety of forms. As notedpreviously, one such form may include a head-mounted display (though thepresent embodiment is not limited thereto). Now with reference to FIG. 6, an example of an apparatus 650 of the present embodiment in the formof such a head-mounted display is depicted therein. The exampleembodiment as shown in FIG. 6 includes a processor 652. Although notvisible in FIG. 6 , a saturation profile, saturation response,saturation profile comparer, and saturation response executor may beinstantiated on the processor 652. The apparatus 650 also includes firstand second sensors 654A and 654B. In addition, the apparatus 650includes a body 656 with the processor 652 and the first and secondsensors 654A and 654B disposed thereon, the body 656 having the form ofa pair of glasses. In the example arrangement shown, the first andsecond sensors 654A and 654B are shown facing outward, slightly toeither side of where a wearer's eyes would be located if the apparatus650 were worn, and facing such that the fields of view of the first andsecond sensors 654A and 654B may correspond at least approximately withthe fields of view of the wearer's eyes. However, this is an exampleonly, and other arrangements may be equally suitable.

In addition, the apparatus 650 as shown in FIG. 5 includes first andsecond displays 646A and 646B, with the first and second displays 646Aand 646B disposed on the body 656 so as to be in front of and proximatea wearer's eyes when the apparatus 650 is worn. Though not required forthe present embodiment, for an arrangement wherein the presentembodiment is incorporated into a head-mounted display as shown in theexample of FIG. 6 such displays may be useful. It is noted that thefirst and second displays 646A and 646B also may be taken as an exampleof additional features that may be incorporated with but that may not berequired by the present embodiment; a wide variety of such features maybe suitable for use with the present embodiment, and it is emphasizedthat the present embodiment is not limited only to the specific elementsshown and described herein.

FIG. 7 is a block diagram of an apparatus that may perform variousoperations, and store various information generated and/or used by suchoperations, according to an embodiment of the disclosed technique. Theapparatus may represent any computer or processing system describedherein. The processing system 790 is a hardware device on which any ofthe other entities, components, or services depicted in the examples ofFIG. 1 through FIG. 6 (and any other components described in thisspecification) may be implemented. The processing system 790 includesone or more processors 791 and memory 792 coupled to an interconnect793. The interconnect 793 is shown in FIG. 7 as an abstraction thatrepresents any one or more separate physical buses, point to pointconnections, or both connected by appropriate bridges, adapters, orcontrollers. The interconnect 793, therefore, may include, for example,a system bus, a Peripheral Component Interconnect (PCI) bus orPCI-Express bus, a HyperTransport or industry standard architecture(ISA) bus, a small computer system interface (SCSI) bus, a universalserial bus (USB), IIC (I2C) bus, or an Institute of Electrical andElectronics Engineers (IEEE) standard 1394 bus, also called “Firewire”.

The processor(s) 791 is/are the central processing unit of theprocessing system 790 and, thus, control the overall operation of theprocessing system 790. In certain embodiments, the processor(s) 791accomplish this by executing software or firmware stored in memory 792.The processor(s) 791 may be, or may include, one or more programmablegeneral-purpose or special-purpose microprocessors, digital signalprocessors (DSPs), programmable controllers, application specificintegrated circuits (ASICs), programmable logic devices (PLDs), trustedplatform modules (TPMs), or the like, or a combination of such devices.

The memory 792 is or includes the main memory of the processing system790. The memory 792 represents any form of random access memory (RAM),read-only memory (ROM), flash memory, or the like, or a combination ofsuch devices. In use, the memory 792 may contain a code. In oneembodiment, the code includes a general programming module configured torecognize the general-purpose program received via the computer businterface, and prepare the general-purpose program for execution at theprocessor. In another embodiment, the general programming module may beimplemented using hardware circuitry such as ASICs, PLDs, orfield-programmable gate arrays (FPGAs).

The network adapter 794, a storage device(s) 795, and I/O device(s) 796,are also connected to the processor(s) 791 through the interconnect 793.The network adapter 794 provides the processing system 2090 with theability to communicate with remote devices over a network and may be,for example, an Ethernet adapter or Fiber Channel adapter. The networkadapter 794 may also provide the processing system 790 with the abilityto communicate with other computers within the cluster. In someembodiments, the processing system 790 may use more than one networkadapter to deal with the communications within and outside of thecluster separately.

The I/O device(s) 796 can include, for example, a keyboard, a mouse orother pointing device, disk drives, printers, a scanner, and other inputand/or output devices, including a display device. The I/O device(s) 796also may include, for example, cameras and/or other imagers adapted toaccept visual input including but not limited to postures and/orgestures. The display device may include, for example, a cathode raytube (CRT), liquid crystal display (LCD), or some other applicable knownor convenient display device. The display device may take various forms,including but not limited to stereo displays suited for use in near-eyeapplications such as head-mounted displays or other wearable devices.

The code stored in memory 792 may be implemented as software and/orfirmware to program the processor(s) 791 to carry out actions describedherein. In certain embodiments, such software or firmware may beinitially provided to the processing system 790 by downloading from aremote system through the processing system 790 (e.g., via networkadapter 794).

The techniques herein may be implemented by, for example, programmablecircuitry (e.g. one or more microprocessors) programmed with softwareand/or firmware, or entirely in special-purpose hardwired(non-programmable) circuitry, or in a combination of such forms.Special-purpose hardwired circuitry may be in the form of, for example,one or more AISCs, PLDs, FPGAs, etc.

Software or firmware for use in implementing the techniques introducedhere may be stored on a machine-readable storage medium and may beexecuted by one or more general-purpose or special-purpose programmablemicroprocessors. A “machine-readable storage medium”, as the term isused herein, includes any mechanism that can store information in a formaccessible by a machine.

A machine can also be a server computer, a client computer, a personalcomputer (PC), a tablet PC, a laptop computer, a set-top box (STB), apersonal digital assistant (PDA), a cellular telephone, an iPhone, aBlackberry, a processor, a telephone, a web appliance, a network router,switch, or bridge, or any machine capable of executing a set ofinstructions (sequential or otherwise) that specify actions to be takenby that machine.

A machine-accessible storage medium or a storage device(s) 795 includes,for example, recordable/non-recordable media (e.g., ROM; RAM; magneticdisk storage media; optical storage media; flash memory devices; etc.),etc., or any combination thereof. The storage medium typically may benon-transitory or include a non-transitory device. In this context, anon-transitory storage medium may include a device that is tangible,meaning that the device has a concrete physical form, although thedevice may change its physical state. Thus, for example, non-transitoryrefers to a device remaining tangible despite this change in state.

The term “logic”, as used herein, may include, for example, programmablecircuitry programmed with specific software and/or firmware,special-purpose hardwired circuitry, or a combination thereof.

The above specification, examples, and data provide a completedescription of the manufacture and use of the composition of theembodiment. Since many embodiments of the embodiment can be made withoutdeparting from the spirit and scope of the embodiment, the embodimentresides in the claims hereinafter appended.

The invention claimed is:
 1. A method, comprising: defining, by aprocessing device, one or more actions to be carried out when asaturation determination is positive; determining by the processingdevice a first saturation level of an object within a first portion of afield of view (FOV) of a sensor at a first point in time; determining bythe processing device a second saturation level of the object within asecond portion of the FOV of the sensor at a second point in time;determining by the processing device that the first saturation level isdifferent than the second saturation level, wherein a difference betweenthe first saturation level and the second saturation level is a positionsaturation determination; and executing, by the processing device, anaction to address a positive saturation determination between the firstsaturation level and the second saturation level.
 2. The method of claim1, wherein the object comprises a human hand occupying a greater amountof one of the first portion of the FOV or the second portion of the FOVthan another of the first portion of the FOV or the second portion ofthe FOV.
 3. The method of claim 1, wherein the first saturation leveland the second saturation level are brightness values and at least oneof the first saturation level or the second saturation level is amaximum brightness level.
 4. The method of claim 1, wherein the firstsaturation level and the second saturation level are brightness valuesand at least one of the first saturation level or the second saturationlevel is a maximum or a minimum brightness value.
 5. The method of claim1, wherein the first saturation level is greater than the secondsaturation level.
 6. The method of claim 1, wherein the processingdevice is further configured to determine a distance of the objectrelative to the sensor, wherein the sensor is coupled to the processingdevice.
 7. The method of claim 1, further comprising a sensor coupled tothe processing device, the sensor configured to sense the object.
 8. Themethod of claim 1, wherein the processing device is configured toanalyze an image or video to determine at least one of the firstsaturation level or the second saturation level.
 9. The method of claim1, wherein the processing device is coupled to a mobile device.
 10. Themethod of claim 1, wherein the processing device is coupled to awearable device configured to be positioned in a viewable positionrelative to an eye of a user.
 11. The method of claim 1, wherein atleast one of the first saturation level or the second saturation levelis representative of a substantially uninformatively uniform brightnesslevel of an image or video.
 12. The method of claim 11, wherein theprocessing device is configured to detect a gesture corresponding to theobject.
 13. The method of claim 12, wherein the gesture is completed atthe second point in time resulting in the second saturation level.
 14. Adevice, comprising: a sensor configured to: sense a first saturationlevel of an object at a first point in time; and sense a secondsaturation level of the object at a second point in time; a memoryconfigured to store: a saturation profile; and a saturation response,wherein the saturation response comprises: navigating a user interface;transmitting data; sending instructions to another device; or executinganother processing function; and a processing device coupled to thesensor and the memory, wherein the processing device is configured to:reference the first saturation level on the saturation profile todetermine a first position of the object within a first portion of afield of view (FOV) of the sensor at the first point in time, whereinthe object occupying the first portion of the FOV is indicative of theobject being a first distance from the sensor; reference the secondsaturation level on the saturation profile to determine a secondposition of the object within a second portion of the field of view(FOV) of the sensor at the second point in time, wherein the objectoccupying the second portion of the FOV is indicative of the objectbeing a second distance from the sensor; and in response to the objectbeing at the first distance at the first point in time and the seconddistance at the second point in time, execute the saturation responsebased on the first saturation level and the second saturation level. 15.The device of claim 14, wherein the first point in time corresponds to abeginning of a gesture and the second point in time corresponds to anend of a gesture.
 16. The device of claim 14, wherein the objectcomprises a hand of a user and a difference between the first positionof the object and the second position of the object corresponds to agesture.
 17. An apparatus, comprising: a sensor configured to: sense amovement of an object; and sense a saturation level of the object; amemory configured to store: a saturation profile; and a saturationresponse, wherein the saturation response comprises: navigating a userinterface; transmitting data sending instructions to another device; orexecuting another processing function; and a processing device coupledto the sensor and the memory, wherein the processing device isconfigured to: identify a gesture based on a change in position of theobject; determine that the saturation level exceeds a thresholdsaturation level; and in response to the movement of the object matchingthe defined movement for the gesture and the saturation level exceedingthe threshold saturation level, execute the saturation response.
 18. Theapparatus of claim 17, wherein the gesture comprises a change in adistance of the object relative to the sensor from a first point in timerecorded by the sensor to a second point in time recorded by the sensor.19. The apparatus of claim 17, wherein the saturation level varies overtime.
 20. The apparatus of claim 17, the processing device is configuredto recognize a first field of view of the sensor and a second field ofview of the sensor to identify the change in position of the object.